draft(qwen4): add hardware-fit runtime and modular artifact path - #1

Stack and intent

This is a draft, stacked review PR for the Qwen3.8/Qwen4-Exp hardware-fit work. It is based directly on the current head of FlashML-org/FreeToken#232 (ad752c9970e0dc3f1b09aeec38235332149336ed). We also continuously reconciled behavior and implementation choices against the parallel FlashML-org/FreeToken#226.

It is intentionally opened against the FlashML-org#232 source branch so the review diff contains only this work. After the upstream Qwen4 direction settles, the useful pieces can be split, reduced, or retargeted for upstream merge.

The goal is to make a 512-expert Qwen3.8-Flash-Next text path technically coherent on a Windows host with 64 GiB RAM and an RTX 5070 12 GB, while keeping formats and failure modes explicit. Feedback and collaboration are welcome, especially on the API boundaries and what should be upstreamed independently.

What is in the draft

• Qwen4 parser/router reconciliation and synthetic sparse-QSA differential oracles.
• A deterministic BF16-to-native-NVFP4 encoder and the selected Qwen4 W4A16 operator path.
• An opt-in GDN split with NVFP4 qkv/z and output projections while b/a remain BF16.
• Native Q3_PLE_32 streaming reader and writer with strict hashes, segment validation, and bounded memory.
• FTEXPERT1 fixed-record expert sidecars plus a real file-backed FileExpertSource integrated with the existing expert cache.
• Mixed resident/file-tier expert orchestration with explicit fail-closed artifact metadata.
• A text-only modular Qwen4 artifact schema and restartable conversion orchestration.
• Windows CUDA TensorMatcher portability fixes required by the production fast-index-copy JIT path.

Measured synthetic evidence

• Stage 6 component closeout: 194 passed, 20 capability/platform skips, 0 failed across unique focused cases.
• Native NVFP4 backend regression: the two previously blocked fast-index-copy cases execute their intended cache/numerical assertions and pass.
• Active NVFP4 CUDA shape matrix: 29 passed.
• QSA/KV differential suite: 22 passed.
• Stage 7 modular artifact/orchestration tests: 124 passed, 4 CUDA skips, plus 54 passed in the Qwen4 parser/model postflight.
• A real-geometry synthetic 512-slot native expert cache allocated 1,419,509,760 payload bytes and passed scatter/fill/teardown checks without a persistent HostBank for the file-tier source.
• Metadata-only dry run classified all 296,475 tensors and reconciled all 206 source shards without transferring any model payload.

All results above are component or synthetic evidence. They are not full-model quality, context-length, inference-throughput, or serving evidence.

Capacity design currently represented

• Active fused runtime-state component: 4,804,403,200 on-disk bytes; 4,802,913,690 persistent representation bytes.
• 48 aligned expert sidecars: 68,149,248,000 bytes.
• Q3 PLE: 22,400,107,520 bytes.
• Known target components: 95,353,758,720 bytes before small metadata/config files.
• Initial file-tier policy: layers 0-5 and 42-47; layers 6-41 resident.

These are exact format/accounting targets, not a claim that the converted real artifact has been built or validated.

Upstream evidence and expert-tier viability update

External testing at the current FlashML-org#232 head (ad752c9) and model revision 7b719225... reports a real 249,999-token prompt under a 262,144 allocation, exact needle retrieval, the upstream FTW/offload path passing, and approximately 15.2-15.6 tok/s on an RTX 4090. This is useful EXTERNAL validation of the upstream full-model/QSA path; it does not validate this custom 12 GB artifact.

That run also reported approximately 69% GPU expert-cache misses with 1,024 slots. This is not directly an SSD-tier miss rate: resident-layer GPU misses are served from HostBank over PCIe, while only misses in the 12 file-tier layers require SSD records. It nevertheless invalidates treating the earlier 1% and 5% SSD sensitivities as expected behavior.

For 12 tier layers, topk=10, and a 2,772,992-byte record, a 69% sensitivity is 82.8 reads/token: 2,296.04 MB/s at 10 tok/s, 3,444.06 MB/s at 15 tok/s, and 4,592.07 MB/s at 20 tok/s. The measured Stage 6 path was warm-buffered and effectively serial, peaking around 938.6 IOPS / 2,602.8 MB/s. Therefore real checkpoint acquisition is now technically blocked pending routing-informed tier selection, another fit-valid placement, or measured bounded asynchronous FileExpertSource behavior.

For eventual matched benchmarking, plain offload is the primary baseline. External matched 4090 results were 14.896 tok/s for offload and 12.753 tok/s for profiled hybrid, so hybrid was 14.4% slower despite the bandwidth profiler recommending it. Hybrid needs matched end-to-end evidence on the target Ryzen 9 5900XT + RTX 5070 machine.

There is also a current storage stop gate: the latest Z: recheck found 252,505,526,272 bytes free, while the retained-source Strategy A needs 309,257,827,893 bytes including the mandatory 64 GiB reserve. Acquisition is therefore short by 56,752,301,621 bytes even before the expert-tier issue is cleared. No unrelated cleanup is implied or authorized by this draft.

Explicitly not done

• No Qwen checkpoint weight payload was downloaded.
• No real model artifact was converted.
• No full model was loaded and no generation/server benchmark was run.
• Real-artifact quality and checkpoint-backed runtime validation remain blocked.
• Cold/unbuffered and genuinely asynchronous FileExpertSource performance remain unresolved. Warm buffered low-miss sensitivities passed, but plausible high-miss operation is not supported by the current synchronous evidence; real acquisition remains blocked.
• Windows JIT validation currently requires a separately recorded compatibility patch for pinned apache-tvm-ffi==0.1.13.post3; upstream TVM-FFI is not represented as working unpatched.

Where review would help most

1. Whether the active NVFP4 selection/config contract is the right minimal extension to the feat(qwen4): support Qwen3.8 Flash Next NVFP4, vision, and Windows serving FlashML-org/FreeToken#232 model path.
2. Whether Q3_PLE_32, FTEXPERT1, and the modular manifest should remain separate components or share more of FTW's indexing machinery.
3. The ExpertSource boundary and the smallest credible path to true asynchronous Windows I/O.
4. Which TensorMatcher/toolchain pieces should be split into independent upstream fixes.
5. How best to converge this with feat(models): add Qwen3.8 Flash Next support FlashML-org/FreeToken#226 and feat(qwen4): support Qwen3.8 Flash Next NVFP4, vision, and Windows serving FlashML-org/FreeToken#232 without duplicating their model integration work.

Local replay validation for this review branch

The ten accepted commits were replayed without conflicts onto the current FlashML-org#232 head. A fresh CPU-focused replay check passed:

124 passed in 23.54s
git diff --check: PASS

Draft only: please do not merge until the upstream base, component boundaries, and later real-artifact authorization are agreed.